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Abstract 

For general repeated measures designs the Wald-type statistie (WTS) is an asymp- 
totieally valid proeedure allowing for unequal eovarianee matriees and possibly non¬ 
normal multivariate observations. The drawbaek of this proeedure is the poor perfor- 
manee for small to moderate samples, i.e., deeisions based on the WTS may beeome 
quite liberal. It is the aim of the present paper to improve its small sample behavior by 
means of a novel permutation proeedure. In partieular, it is shown that a permutation 
version of the WTS inherits its good large sample properties while yielding a very 
aeeurate finite sample eontrol of the type-I error as shown in extensive simulations. 
Moreover, the new permutation method is motivated by a praetieal data set of a split 
plot design with a faetorial strueture on the repeated measures. 

Keywords: Permutation Tests; Longitudinal Data; Quadratie Forms; Repeated Measures. 


* University of Ulm, Institute of Statisties, Germany 
email: sarah.friedrieh@uni-ulm.de 

** University Medieal Center Gottingen, Institute of Medieal Statisties, Germany 


I 



1 Motivation and Introduction 


In many experiments in the life, soeial or psyehologieal seienees the experimental units 
(e.g. subjeets) are repeatedly observed at different oeeasions (e.g. at different time points) 
or under different treatment eonditions. This leads to eertain dependeneies between obser¬ 
vations from the same unit and results in a more eomplieated statistieal analysis of sueh 
studies. In the eontext of experimental designs, the repeated measures are eonsidered as 
levels of the sub-plot faetor. If several groups are observed, these are eonsidered as levels 
of the whole-plot faetor. Typieal questions in this setting eoneem the investigation of a 
group effeet, a non-eonstant effeet of time or different time profiles in the groups. Classi- 


eal repeated measures models, where hypotheses are tested with Hotelling’s (Hotelling 


1931|) or Wilks’s A (Wilks, 1932), assume normally distributed observation veetors and 


a eommon eovarianee matrix for all groups, see e.g. the monograph of Davis (2002). In 
medieal and biologieal researeh, however, the assumptions of equal eovarianee matriees 
and multivariate normally distributed outeomes are often not met and a violation of them 
may inflate the type-I error rates, see the eomments in |Xu and Cui| ( |2008| ), |Suo et aL] ( |2013| ) 
orlKonietsehke et aL](|2015|). Therefore, other proeedures have been developed for repeated 


measures whieh are based on eertain approximation teehniques (Geisser and Greenhouse 


(1958 

); Greenhouse and Geisser (1959); Huynh and Eeldt (1976); 

Eeeoutre (1991); Ke- 

selman et al. (2000); Werner (2004); Ahmad et al. 

2008) 

Brunner 

(2009); Brunner et al. 

(2009 

); Ken ward and Roger (2009); Brunner et al. 

(2012 

); Chi et al. (2012); Pauly et al. 


only diseuss methods for speeifie models whieh are also asymptotieally only approxima¬ 
tions, i.e., they do not even lead to asymptotie exaet tests. Another possibility is to apply a 
speeifie mixed model in the GEE eontext, see e.g. the text books by Verbeke and Molen- 
berghs (2009, 2012). These methods require that the data stems from a speeifie exponential 
family. An exeeption is given by the multivariate Wald-type test statistie (WTS) whieh is 
asymptotieally exaet. However, it is well known that it requires large sample sizes to keep 


the pre-assigned type-I error level, see e.g. Brunner (2001), Pauly et al. (2015a) and Koni- 


etsehke et al. (2015). 


To improve the small sample behavior of the WTS in a MANOVA setting, Konietsehke et 


aL] ( |2015| ) proposed different bootstrap teehniques. Another possibility would be to apply 


permutation proeedures. It is well known that permutation tests are finitely exaet under 
the assumption of exehangeability, see e.g. Brombin et al. (2013|),|Pesarin (|2001 ), Mielke 


and Berry (2007|) or Pesarin and Salmaso|(2010a|b[ 2012[) for examples. In most of these 


examples, however, permutation tests are only applied in situations where the null distri¬ 
bution is invariant under the eorresponding randomization group. A modified permutation 
proeedure may also be applied in situations where this invarianee does not hold, see e.g. 


Janssen and Pauls (2003), Janssen (2005), Omelka and Pauly (2012), Chung and Romano 


( |2013[ ) and [Pauly et al.| ( |2015a| ). The main idea in these papers is to apply a studentized test 
statistie and to use its permutation distribution (based on permuting the pooled sample) for 
ealeulating eritieal values. This leads to partieularly good finite sample properties even in 
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case of general faetorial designs with fixed faetors, see [Pauly et al. (2015a). It is the aim of 
the present paper to extend the eoneept of permuting all data to the eontext of longitudinal 
data in general (not neeessarily normal and homoseedastie) split plot designs. Applied to 
the WTS this generalizes the results of [Pauly et al.| ( |2015al ) and leads to astonishingly ae- 
eurate results despite the dependeneies in repeated measurements data. 


The methodology derived in this paper is motivated by the following data example on 
the O 2 eonsumption of leukoeytes. To examine the breathability of leukoeytes, an exper¬ 
iment with 44 HSD-rats was eondueted. A group of 22 rats was treated with a plaeebo, 
while the other 22 rats were treated with a substanee supposed to enhanee the humoral 
immunity. 18 hours prior to the opening of the abdominal eavity, all animals reeeived 2.4 
g sodium-easeinat for the produetion of a peritoneal exudate rieh on leukoeytes. In order 
to obtain a suffieient amount of material the peritoneal liquid of 3-4 animals was mixed 
and the leukoeytes therein were rehashed in an experimental bateh. One half of the exper¬ 
imental bateh was mixed with inactivated staphyloeoeei in a ratio of 100:1, the other half 
remained untreated and served as a eontrol. Then, the oxygen eonsumption of the leuko¬ 
eytes was measured with a polarographie eleetrode after 6, 12 and 18 minutes, respeetively. 
For eaeh group separately, 12 experimental batehes were earried out. The means over the 
experimental batehes in both treatment groups are listed in Table 

Table 1: Mean oxygen consumption of leukocytes in the presence and absence of inacti¬ 
vated staphylococci. 


Mean O 2 -Consumption {p£\ 


Staphy 

With 

ococci 

Without 


Time [min] 

6 12 18 

Time [min] 

6 12 18 

Placebo 

1.618 2.434 3.527 

1.322 2.430 3.425 

Verum 

1.656 2.799 4.029 

1.394 2.57 3.677 


Questions of interest in this example coneern the effeet of the whole-plot factor ’treat¬ 
ment’, the effeet of the sub-plot factors ’staphylococci’ and ’time’ as well as interaetions 
between these effeets. We note that the empirieal 6x6 eovarianee matriees of the two 
groups appear to be quite different (see the supplement for details). This also motivates to 
include unequal covariance matriees in our model. For sueh experimental designs proee- 
dures are derived in this paper that lead to good small sample eontrol of the type-I error 
while being asymptotieally exaet. 

The paper is organized as follows. The underlying statistieal model is deseribed in See- 
tion[^ where we also introduee the Wald-type (WTS) as well as the ANOVA-type statistie 
(ATS) and state their asymptotie behavior. In Seotion|^ we deseribe the novel permuta¬ 
tion proeedure used to improve the small sample behavior of the WTS. Afterwards, we 
present the results of extensive simulation studies in Seetion Q analyzing the behavior of 
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the permuted test statistie in different simulation designs with eertain eompetitors. Addi¬ 
tional simulation results have also been run for several other resampling sehemes. They 
did not show a better performanee than the permutation proeedure and are only reported 
in the supplementary material, where also various power simulations ean be found. The 
motivating data example is analyzed in detail in Seetion The paper eloses with a brief 
diseussion of our results in SeetionAll proofs are given in the supplementary material. 


2 Statistical Model, Hypotheses and Statistics 


2.1 Statistical Model and Hypotheses 


To establish the general model, let 


ik 


• • • : ^ 1, . . . , Cl, k 1, . . . , 7li 


( 2 . 1 ) 


denote independent random veetors with distribution Fj,expeetation = {fxn,..., ^iu)' = 
E{Yii) and eovarianee matrix Vi = Cov{Yii) > 0 in treatment group i. We do not as¬ 
sume any special structure of the eovarianee matrix Vi whieh may even be different be¬ 
tween groups 1 < i < a. Note that we also allow the number of time points to differ 
between groups. The most eommon ease where ti = f for alH = 1,..., a is thus a speeial 
ease of model ( |2.1| ). Here the time points G N are fixed. For eonvenienee, we eolleet the 
observation veetors Yik in 


Y = (Y’„...,Y'J, Y.^(Y;„...,YiJ'. (2.2) 

In this set-up, hypotheses are formulated as IIq : Hp = 0, where p = {p[, ..., p'ff 
denotes the veetor of all expeetations pis = EiYus), f = l,...,a; s = l,..., ti and H is 
a suitable eontrast matrix, i.e., its rows sum up to zero. Examples of H are presented in 
Seetion m 

Throughout the paper, we will use the following notation. We denote by It the t- 
dimensional unit matrix and by Jt the t x t matrix of I’s, i.e., Jt = 1*1^, where 1^ = 
(1,..., 1)' is the f-dimensional eolumn veetor of I’s. Furthermore, let Pt = It — \ Jt 
denote the f-dimensional eentering matrix. By © and © we denote the direet sum and the 
Kroneeker produet, respeetively. 

An estimator of p is given by 


^ 1 


r. = (y;., ..., y'j for Y,. = (ni,..., Vtj, n. = - Vks, 5 = 1,..., 

r) • f ^ 


Hi 


k=l 


and the eovarianee matrix Vi in treatment group i is estimated by the sample eovarianee 
matrix 


Ui 


V^ = —— J2^Y,k - Yff{Y,k - Yi.)\ i = 1 , 

Tli i 


,a. 


k=l 
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Let N = J2i=i denote the total number of subjects in the trial, T = YTi=i the 
total number of time points and N = Yl°i=i the total number of observations. Then the 
asymptotic results are derived under the following two assumptions: 

(1) ^ G (0,1) as min(ni,..., -)■ cx), 

(2) supiE(||l^iiin < OO. 


2.2 Statistics and Asymptotics 


We consider two commonly used test statistics for repeated measures and multivariate data. 
First, the so-called ANOVA-type statistic (ATS), introduced in|Brunner|(|2001[), is given as: 


Qn = NYH'iHH'yHY = NY'tY., 


(2.3) 


where (■)“ denotes some generalized inverse. Note that the test statistic docs not depend 
on the special choice of the generalized inverse. Its asymptotic distribution is established 
in the next theorem. 


Theorem 2.1 Under the null hypothesis ifg : Hp 
cally, the same distribution as the random variable 


0, the ATS in (2.3) has, asymptoti- 




; -AX. 7 e 


2=1 S = 1 


where Xis Xi the weights Xis are the eigenvalues ofTHfor S = 0“=]^ Kj i. 
Moreover, for local alternatives Tp = ■ Tu,u G Mf, it holds that the ATS has, 

asymptotically, the same distribution as Z'TZ, where Z ~ A^(rz, S). If additionally 
S > 0, the ATS has the same distribution as a weighted sum o/x^(5) distributed random 
variables, where the weights are again the eigenvalues Xis (^^d 5 = 


Since the Xis are unknown, the result cannot be applied directly. Nevertheless, Brunner 
(20011 proposed to approximate the distributio n of X by th e distribution of a scaled 


distribution, i.e., by g ■ where X^y ~ xt (Box 1954). The constants g and u are 


estimated from the data such that the first two moments of X and g ■ X^, coincide (Box 


1954). This leads to approximating the statistic 


Fn = 


^ y'ty 


tr(TS) 


(2.4) 


by an F(z/, oo)-distribution with estimated degree of freedom z> = tr^(TS)/tr(TS)^, 
where H = N 0“^^ The corresponding ATS test pats = IIQtv > Faiv,oo)}, 


5 



















where -Fq(z/, cx)) denotes the (1 — Q!)-quantile of the F(z/, cx))-distribution, leads to eonsis- 
tent test deeisions for fixed alternatives. However, it is in general no asymptotic level a test 
under the null hypothesis, which is a severe drawback of this procedure. Thus, we discuss 
a second statistic, the so-called Wald-type statistic (WTS) given as 


= NY'h'{H%H') HY . 


(2.5) 


Here (iTSiT')+ denotes the Moore-Penrose inverse of {HUH'). In order to test the 
general linear hypotheses : H/j, = 0 critical values are taken from the asymptotic 
distribution of Qn under the null hypothesis stated below. 


Theorem 2.2 Under the null hypothesis Hq : Hp = 0, the WTS in ( |2.5[ ) has, asymp¬ 
totically, a central x^f distribution with f = rank{H). The corresponding test is given by 
Twts = ^{Qn > where Xpi-a denotes the (1 — a)-quantile of the x} distribu¬ 

tion. This test is an asymptotic level a test and is consistent for general fixed alternatives 
Hp 0. Moreover, for local alternatives Hp = it holds that Qn has 

asymptotically a non-central x}{^) distribution where S = {Hu)'{HUH'Hu. This 
implies that Eh fi^wTs) P{Z > x}y-a) ^dh Z X}i5). 


Although pwTS possesses these nice asymptotic properties, it is well-known that very large 
sample sizes n* are necessary to maintain the pre-assigned level a using quantiles of the 


limiting -distribution, see Konietschke et al. (20151, Pauly et al. (2015a) and Brunner 


(|2001 ) as well as Tablej^below. This leads to a limited applicability of the WTS in practice. 


To accept the need for a novel procedure, we investigate the accuracy of the two test 
statistics in a one sample repeated measure design with n subjects and t repeated measures 
Yks. The null hypothesis ■ {pi = ■ ■ ■ = Pt} = {PtP = 0},p = {pi, ..., pt)'ts 
considered and the components of are selected as standardized log-normally distributed 
random variables, i.e.. 


yr _ ^ks dS{^ks) 

for i.i.d. log-normally distributed e^s, k = 1,... ,n and s = 1,... ,t. The results are 
displayed in Tablewhere the simulated type-I error rates of the WTS and ATS are given. 
It is readily seen that the test based on the WTS considerably exceeds the nominal level of 
5%, while the ATS leads to rather conservative decisions. 
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Table 2: Simulated type-1 error rates (10000 simulations) in a repeated measures design 
with n = 10, 20, 50,100 individuals and t = 4, 8 repeated measures. The ATS is compared 
to the upper 5% quantile of the F{i), oo)-distribution, the WTS to the upper 5% quantile of 
the Xt-i-distribution. 



Type-I error rates (a = 0.05) 


ATS: F-quantile 

WTS: x^-quantile 

n 

t = 4 

t = 8 

t = 4 

t = 8 

10 

0.025 

0.012 

0.223 

0.776 

20 

0.026 

0.014 

0.126 

0.388 

50 

0.030 

0.021 

0.081 

0.166 

100 

0.035 

0.025 

0.067 

0.111 


Thus, to enhance the small sample properties of the above tests we have compared dif¬ 
ferent resampling approaches in an extensive simulation study, presented in Section of 
the supplementary material Friedrich et al. ( 2016| ). Surprisingly, the best procedure turned 
out to be a permutation technique that randomly permutes the pooled univariate observa¬ 
tions without taking into account the existing dependencies for calculating critical values. 
This at first sight counter-intuitive method is motivated from |Konietschke and Pauly] ( |2014[ ), 
where a similar approach has been applied in the paired two-sample case. Moreover, the 


current procedure generalizes the permutation test on independent observations by Pauly et 


al. (2015a) and implemented in the R package GFD ([Friedrich et al.f 2015| to the case of 


repeated measures and multivariate data. The details are explained in the next section. 


3 The Permutation Procedure 


Let = 7r(F] 


111 ) 


YanataY = (>"iTi) ■ • ■ ) ^enotc a fixed but arbitrary permu- 


denotes the 


tation of all N elements of Y in ( |2.2| ), i.e., vr G 5^. In this notation 
(i, k, s)—component of the permuted vector Y. Furthermore, let Y denote the vector of 
the means under this permutation and S = *^he empirical covariance matrix 

of the permuted observations. 

It is obvious, that Y and Y"" only have the same distribution, if the components of Y 
are exchangeable. However, this is not the case in general two- and higher way layouts. 


even in the case of independent observations, see e.g. Huang et al. (2006). Following the 


approach of |Neuhaus] ( |1993| ), [Janssen] ( |1997[[2005| ), |0melka and Pauly | ( |2012| ), [Chung and 
Romano ( [2013 ) and Pauly et al. (2015a) in the case of independent observations, the idea 
is to studentize the statistic \/NY“, and consider its projection into the hypothesis space, 
resulting in the WTS of the permuted observations, namely 


= N{Y"yH'{H^"HYHY" 


(3.1) 
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In the sequel we will denote QJ^ as the WTPS. Note, that the question how to permute is 
more involved here than in the ease of independent univariate observations. A heuristie 
reason why the above approaeh might work is as follows: Uneonditionally, all permuted 
eomponents possess the same mean. Thus, when multiplied by a eontrast matrix the per¬ 
muted means veetor always mimics the null situation, i.e., HE{Y, ) = 0 always holds. 
In particular, it can be shown that the conditional distribution of the WTPS in ( |3.1| ) 
always approximates the null distribution of Qjv in ( |2.5| ) in the general repeated measures 
design under study; thus leading to an asymptotically valid permutation test. This result is 
formulated in the following theorem: 


Theorem 3. 1 The studentized permutation distribution of in 53) conditioned on 
the observed data Y weakly converges to the central xj distribution in probability, where 
f = rank{H). 

Remark 3.1 Theorem |3. J [ states that the permutation distribution asymptotically provides 
a valid approximation of the null distribution of the test statistic Qn in ( |2.5| ). To be con¬ 
crete, this means that for any underlying parameters p, G and G Hq{H) with 
HPq = 0 we have convergence in probability 

sup \P^{Qn < x\Y) - P^j.^iQN < a;)| -)■ 0. (3.2) 

Here, Pij.{Qn < x) and P^(Qjf < x|l^) denote the unconditional and conditional dis¬ 
tribution function of Qn nnd respectively, under the assumption that p is the true 
underlying parameter. 


Remark 3.2 A Wald-type permutation test is obtained by comparing the original test 
statistic Qn with the (1 — a)-quantile c\_^ of the conditional distribution of the WTPS 
Q'lf given the observed data Y, i.e., pwtps = '^{Qn > cl_^}. Theorem \3.1\ implies 
that this test asymptotically keeps the pre-assigned level a under the null hypothesis and is 
consistent for any fixed alternative Hp f 0, i.e., it has asymptotically power 1. Moreover, 
it has the same asymptotic power as the WTS for local alternatives Hp = -^ ■ r', i-e., it 

holds that Eh^ (pwtps) P{Z > xjy-J ^^th Z Xf(^) as in Theorem 


2.2 


It follows that the permutation test and the classical Wald-type test are asymptotically 
equivalent and that both have the same local power under contiguous alternatives. In par¬ 
ticular the asymptotic relative efficiency of the WTPS compared to the classical WTS is 
1. Moreover, the permutation test based on is finitely exact if the pooled data Y are 
exchangeable under the null hypothesis. In comparison, the ATS also leads to a consistent 
test for fixed alternatives but does not provide an asymptotic level a test since it is only an 
approximation. 


We note, that the proof given in the supplement to this paper indicates that the given per¬ 
mutation technique does not work in case of the ATS. In particular, a permutation version 
of the ATS would also possess a weighted x^-limit distribution but with different weights, 
say Xis, due to an incorrect covariance structure. 
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Remark 3.3 Our general framework ( |2.1[ ) allows for the treatment of different impor¬ 
tant factorial designs in the context of multivariate repeated measures data analysis. As in 
Pauly et al. ^20 15a) the idea is to accordingly split the indices in subindices and to choose 


an appropriate hypothesis matrix H. Examples of different cross-classified and hierarchi- 


cally nested designs are discussed in Section 4 of Konietschke et al. 

(2015). For repeated 

measures, examples are given in Sections!^andl^ below as well as in 

Brunner (2001). 


4 Simulations 

In order to investigate the small sample behavior of the WTPS, we present extensive simu¬ 
lation results for different designs and covariance structures. The procedure is analyzed in 
different settings with regard to maintaining the pre-assigned type-I error rate (a = 5%). 
The results for the WTPS are compared to the asymptotic quantiles of the ATS (F-quantile) 
and the WTS -quantile). 


4.1 Data Generation 


For our simulation studies, we simulated a split plot design which, in the context of longi¬ 
tudinal data, is a design with a groups, rii subjects in group i and E = t repeated measures 
^iksi ^ 1,..., t. Let 

Yik = (Ljfci,..., Yikt)' = Pi Biklt + VJ Sik, 

with Pi = E{Yii),i = 1,..., a, and let Bik ~ iV(0, af) denote independent additive sub¬ 
ject effects. The i.i.d. random vectors = (ejfci,..., Cikt) were generated from different 
standardized distributions by 

_ ^iks E(riks') 

^JVar{eiks) ’ 

where iiks denote i.i.d. normal, exponential or log-normal random variables. 

A simulation setting with a = 3 groups and f = 4, 8 repeated measures was considered. 
The null hypotheses investigated are 


(1) The hypothesis of no time effect T 

H^{T) : Fi = ■ ■ ■ = Ft or equivalently H^p = 0, 


(2) The hypothesis of no group x time interaction effect GT 


Ht(GT) 


HgtP 


^ Pii ~ Pi- ~ P-i + p.. ^ 
y pat ~ Pa- ~ P-t T P-- j 


where Ht = ® Pt and Hqt = Pa® Pt- 


0, 
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We considered balanced as well as unbalanced designs for the n = (ni, ^ 2 ,^, 3 ) subjects 
in group 1-3, respectively. The simulated numbers of subjects were = (30, 20,10), 

(10, 20, 30) and = (15,15,15). Furthermore, we simulated three different covariance 
structures Vi 

Setting 1: Vj = Jt for alH = 1, 2, 3 

Setting 2: Vi = diag{a\, ..., a^) with = s for f = 4 and al = ^/s for f = 8 

Settings: Vi = , (pi,P 2 ,P 3 ) = (0.6, 0.5, 0.4) fori = 1,2,3 

V / l,j<t 


In Setting 1 and 2 the covariance structures are the same for all groups, whereas in Setting 3 
we have an autoregressive covariance structure with different parameters for the different 
groups. Moreover, we simulated block effects with different variances af G {0,1,2}. 
However, since the results were almost identical, we here only report the case af = 0. All 
simulations were conducted with 10,000 simulation and 1,000 permutation runs. 


4.2 lype-I error rates 

The resulting type-I error rates for the hypotheses of no time effect T and no group x time 
interaction GT are displayed in Tables]^ and Q respectively. 

It is obvious that the tests based on the WTS considerably exceed the nominal level 
for small sample sizes. This behavior becomes worse with an increasing number of re¬ 
peated measurements and when testing the interaction hypothesis. In some cases, the WTS 
reaches an empirical type-I error rate of almost 50% when testing the GT -interaction. This 
means that its accuracy is no better than flipping a coin. The ATS, in contrast, keeps the 
pre-assigned level a pretty well for normally distributed observations, even for small sam¬ 
ple sizes. With an increasing number of repeated measurements and/or non-normal data, 
however, the ATS leads to quite conservative decisions. Furthermore, the ATS leads to 
slightly conservative decisions when testing the interaction hypothesis, even with normally 
distributed data. The WTPS is reasonably close to the pre-assigned level a in most situ¬ 
ations, even under non-normality and for testing the interaction hypothesis. Despite the 
dependencies in longitudinal data, the permutation procedure greatly improves the behav¬ 
ior of the WTS in small sample settings. However, when testing the interaction hypothesis 
for / = 8 repeated measurements the WTPS shows a more or less conservative behavior in 
Setting 3 combined with and a slightly liberal behavior for Setting 3 with 
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Table 3: Results of the simulation studies for the hypothesis of no time effect. 

normal distribution 


T 

t = 4 

t = 8 

Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 



0.046 

0.085 

0.050 

0.040 

0.177 

0.050 

1 


0.046 

0.086 

0.048 

0.040 

0.177 

0.052 



0.050 

0.078 

0.051 

0.043 

0.135 

0.052 


nT) 

0.051 

0.085 

0.050 

0.042 

0.177 

0.051 

2 


0.052 

0.086 

0.051 

0.043 

0.177 

0.052 



0.053 

0.077 

0.051 

0.041 

0.135 

0.052 



0.046 

0.092 

0.052 

0.044 

0.198 

0.062 

3 


0.051 

0.080 

0.045 

0.048 

0.155 

0.042 



0.051 

0.078 

0.053 

0.048 

0.136 

0.054 


log-normal distribution 


Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 



0.032 

0.094 

0.051 

0.021 

0.198 

0.047 

1 


0.031 

0.090 

0.052 

0.020 

0.198 

0.046 


77, (,y) 

0.031 

0.089 

0.051 

0.021 

0.186 

0.048 


77,(1) 

0.040 

0.110 

0.067 

0.022 

0.207 

0.053 

2 


0.040 

0.107 

0.067 

0.022 

0.203 

0.051 


77(y) 

0.042 

0.107 

0.070 

0.024 

0.197 

0.057 



0.033 

0.101 

0.057 

0.024 

0.221 

0.064 

3 


0.037 

0.090 

0.053 

0.033 

0.190 

0.048 


77(y) 

0.036 

0.092 

0.057 

0.031 

0.191 

0.062 


exponential distribution 


Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 


77(1) 

0.045 

0.090 

0.048 

0.034 

0.194 

0.051 

1 

77(^) 

0.046 

0.096 

0.053 

0.032 

0.191 

0.048 


77(y) 

0.046 

0.086 

0.054 

0.034 

0.151 

0.050 



0.048 

0.093 

0.054 

0.035 

0.194 

0.052 

2 


0.050 

0.101 

0.060 

0.034 

0.193 

0.051 


77(1*) 

0.050 

0.088 

0.058 

0.036 

0.154 

0.051 


77(1) 

0.049 

0.098 

0.055 

0.042 

0.218 

0.066 

3 

77 )^) 

0.050 

0.090 

0.049 

0.046 

0.173 

0.045 


77 )^) 

0.050 

0.087 

0.055 

0.042 

0.153 

0.056 
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Table 4: Results of the simulation studies for the hypothesis of no group x time interaction. 

normal distribution 


GT 

t = 4 

t = 8 

Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 



0.049 

0.135 

0.046 

0.033 

0.432 

0.051 

1 


0.053 

0.142 

0.052 

0.034 

0.433 

0.050 



0.048 

0.126 

0.049 

0.039 

0.366 

0.051 


77 ,( 1 ) 

0.053 

0.132 

0.050 

0.038 

0.429 

0.052 

2 

n(^) 

0.053 

0.141 

0.054 

0.038 

0.431 

0.050 


n(^) 

0.050 

0.122 

0.052 

0.040 

0.366 

0.050 



0.054 

0.141 

0.050 

0.040 

0.465 

0.065 

3 


0.053 

0.135 

0.045 

0.049 

0.393 

0.037 


n(^) 

0.051 

0.126 

0.049 

0.045 

0.363 

0.053 


log-normal distribution 


Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 



0.024 

0.121 

0.047 

0.012 

0.426 

0.053 

1 

n(^) 

0.022 

0.128 

0.053 

0.013 

0.431 

0.051 


n(^) 

0.024 

0.118 

0.048 

0.012 

0.406 

0.051 


77,(1) 

0.025 

0.129 

0.051 

0.014 

0.427 

0.054 

2 

n(^) 

0.026 

0.130 

0.054 

0.013 

0.432 

0.052 


n(^) 

0.023 

0.120 

0.050 

0.013 

0.403 

0.052 



0.029 

0.133 

0.050 

0.020 

0.457 

0.062 

3 

n(^) 

0.028 

0.121 

0.045 

0.024 

0.399 

0.036 


n(^) 

0.028 

0.122 

0.049 

0.020 

0.408 

0.053 


exponential distribution 


Cov. Setting 


ATS 

WTS 

WTPS 

ATS 

WTS 

WTPS 


77(1) 

0.043 

0.146 

0.054 

0.024 

0.442 

0.054 

1 

n(^) 

0.041 

0.148 

0.054 

0.024 

0.443 

0.050 


77 )^) 

0.036 

0.122 

0.047 

0.028 

0.397 

0.054 



0.048 

0.151 

0.059 

0.027 

0.444 

0.057 

2 


0.042 

0.153 

0.059 

0.025 

0.448 

0.052 


77 )^) 

0.034 

0.121 

0.048 

0.029 

0.397 

0.055 


77(1) 

0.047 

0.155 

0.061 

0.032 

0.473 

0.068 

3 

77 )^) 

0.043 

0.140 

0.049 

0.042 

0.406 

0.037 


77 )^) 

0.037 

0.122 

0.047 

0.041 

0.402 

0.058 
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The simulations show a clear advantage of the permutation procedure as compared to 
the approximation of the Wald-type statistic. The WTPS controlled the 5% level in 
most situations, even under non-normality, i.e., in situations where the ATS may lead to 
quite conservative decisions. 


4.3 Additional simulation results 


We note that additional simulations for the type-I error can be found in the supplementary 
material to this paper. There we have compared the above methods with other resampling 
schemes such as the bootstrap procedures described in Konietschke et al. ( 2015[ ). Of all 
procedures analyzed in the simulations, the permutation procedure produced the best re¬ 
sults. 


4.3.1 Quality of the approximation 

In the following, we denote by Fjv the distribution function of Qat under by F the distri¬ 
bution function of the limiting X/-distribution under FIq and by n the distribution function 
of the WTPS under ifo- Wecannow define KQS = supo. 9 <t<o ,99 as well 

as KQS'^ = supo. 9 <j<o ,99 ~ order to compare the distance between 

the quantile function F^^ and the limiting quantile function F~^ (KQS) with the distance 
between and (F^) the quantile functions of the test statistic and its permuted ver¬ 
sion (KQS^), respectively. We have calculated these distances for all simulation settings 
described above. Detailed results can be found in Section[^of the supplementary material. 
It turned out that KQS'" is always smaller than KQS, i.e., the approximation provided by 
the permutation procedure is considerably better than the asymptotic approximation for 
all simulation settings considered. In our simulations, KQS ranged from 1.991 to 48.11 
with a median distance of 9.179, whereas KQS" ranged from 0.1049 to 7.618 with a me¬ 
dian value of 0.8948. Figure [^exemplarily shows the plots of the corresponding quantile 
functions for one of the simulation scenarios. 


4.3.2 Large sample behavior 

In this section, we analyze the large sample behavior of the WTS and WTPS. We consid¬ 
ered only normally distributed random variables with covariance structure Setting 2 for an 
unbalanced = (30,20,10)) as well as a balanced (n^^'> = (15,15,15)) design with 
t = 4, 8 time points. The sample size was increased by adding 6 I 3 to the above sample size 
vectors for 6 = 0, 20,..., 200. The results for the type-I error under the null hypothesis of 
no interaction and covariance setting 2 are presented in Figure]^ The behavior of the WTS 
improves with growing sample size but even for 115 individuals in all groups, the WTS 
still exceeds the nominal level. The WTPS, in contrast, is rather close to the pre-assigned 
level even for small sample sizes. 
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Quantile Function 


Figure 1: Quantile funtions of the WTS, WTPS and the corresponding -distribution in the 
balanced simulation setting with log-normally distributed data, t = 8, covariance matrix 
setting 2 and under the null hypothesis of no interaction. 
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Figure 2: Type-1 error rates under the interaction hypothesis for the WTS and the WTPS, 
where sample size was increased by adding 6 I 3 , b = 0, 20,, 200 to the sample size 
vectors in a balanced (lower panel) and unbalanced (upper panel) design with t = A 
(left panel) and t = 8 (right panel) time points under covariance setting 2, i.e., Vi = 
diag(a‘l ,..., ) with = s for t = A and = ^/sfor t = 8. 
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4.3.3 Power 


The power simulations are explained in detail in Section of the supplementary material 
to this paper. Since the WTS turned out to test on different a—levels (see the simulation 
results under the null hypothesis), we have excluded it from the analyses. We addition¬ 
ally considered the approximation described by Lecoutre| ( |1991| ) as well as Hotelling’s 
(Hotelling 19311. It turns out that the ATS has the highest power for normally dis¬ 


tributed data, performing slightly better than the WTPS. For log-normally distributed data, 
the WTPS has larger power than the other methods and it is the only method controlling 
the type-I error correctly. 


5 Application: Analysis of the Data Example 


Finally, we analyze the data example on oxygen consumption of leukocytes in the presence 
and absence of inactivated staphylococci. In this setting we wish to analyze the effect of 
the whole-plot factor ’treatment’ (factor A, Placebo/Verum, a = 2) as well as the sub-plot 
factors ’staphylococci’ (factor B, with/without, 6 = 2) and ’time’ (factor T, 6/12/18 min, 
t = ti = 3,i = 1,..., ab). We are also interested in interactions between the different 
factors. The mean values of the data are given in Tablein Section 


In the analysis we compared the three tests discussed above: The ATS in ( |2.4| ) is 
compared to the corresponding F(z>, cx))-quantile, the WTS in ( |2.5| ) to the asymptotic x}- 
quantile as well as the quantile obtained by the permutation procedure (WTPS). The seven 
different null hypotheses of interest about main and interaction effects can be tested by 
choosing the related hypotheses matrices. Here, we have chosen H a = Pa® 


Hn = U' 


and Ht = -11 

t t ^ a o- 




b 

Pt for testing the main effect of the three 


factors A, 5, and T. For the interaction terms we used the matrices Hat = Pa^ 
Hab = Pa®Pb® j'i-'t and Hbt = ®Pb® Pu and H abt = Pa®Pb 
resulting p-values of the analysis are presented in Table [14} 


-ins 

^Pf 


)Pu 

The 


Table 5: Results of the analysis of the O 2 consumption data. 



ATS 

WTS 

WTPS 

A 

0.001 

0.001 

<0.001 

B 

<0.001 

<0.001 

<0.001 

T 

<0.001 

<0.001 

<0.001 

AB 

0.110 

0.110 

0.118 

AT 

0.009 

<0.001 

0.001 

BT 

0.094 

0.115 

0.141 

ABT 

0.117 

0.116 

0.156 
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For this example all tests under eonsiderations lead to similar eonelusions: Eaeh faetor 
(treatment, staphyloeoeei and time) has a signifieant influenee on the O 2 eonsumption of 
the leukoeytes. Moreover, there is a signifieant interaetion between treatment and time. 


6 Conclusions and Discussion 


In this paper, we have generalized the permutation idea of Pauly et al. (2015a) for indepen¬ 
dent univariate faetorial designs to the ease of repeated measures allowing for a faetorial 
strueture. Here, the suggested permutation test is asymptotieally valid and does not require 
the assumptions of multivariate normality, equal eovarianee matriees or balaneed designs. 
It is based on the well-known Wald-type statistie (WTS) whieh possesses the benefieial 
property of an asymptotie pivot while being applieable for general repeated measures de¬ 
signs. Sinee it is well known for being very liberal for small and moderate sample sizes, 
we have eonsiderably improved its small sample behavior under the null hypothesis by a 
studentized permutation teehnique. For univariate and independent observations the idea 
of this teehnique dates baek to Neuhaus ( |1993 ) and Janssen ( |1997 ) and has reeently been 
eonsidered for more eomplex designs in independent observations by [Chung and Romano 
( |2013[ ) and [Pauly et al.| ( |^15a| ). 

In addition, we have rigorously proven in Theorem [3.1 [ that the permutation distribution 
of the WTS always approximates the null distribution of the WTS and ean thus be applied 
for ealeulating data-dependent eritieal values. In partieular, the result implies that the eorre- 
sponding Wald-type permutation test is asymptotieally exaet under the null hypothesis and 
eonsistent for fixed alternatives while providing the same loeal power as the WTS under 
eontiguous alternatives. 


Moreover, our simulation study indieated that the permutation proeedure showed a very 
aeeurate performanee in all designs under eonsideration with moderate repeated measures 
(t=4) and homoseedastie or slightly heteroseedastie eovarianees. Only in the ease of a 
larger number of repeated measurements (t=8) the WTPS showed a more or less liberal 
(eonservative) behavior when testing the interaetion hypothesis in an unbalaneed design. 
However, all other eompeting proeedures eonsidered in the paper and the supplementary 
material did not perform better in these situations. 

Roughly speaking, the good performanee of the WTPS for finite samples may be ex¬ 
plained by a better approximation of the underlying distribution of the WTS by the per¬ 
mutation distribution as eompared to the x^-distribution. This eould be seen elearly in the 
distanees between the quantile funetions. 
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where we applied all considered approaches. 


* University of Ulm, Institute of Statistics, Germany 
email: markus.pauly@uni-ulm.de 

** University Medical Center Gottingen, Institute of Medical Statistics, Germany 


21 



8 Proofs 


Proof of Theorem 


2.1 


First note that T = T' as well as = T. Let T/j, = ^ for 


ly G i.e., for r/ = 0 we are working under Hq. It holds that '/N{Y. — fi) has, 
asymptotieally, a multivariate normal distribution with mean u and eovarianee matrix S. 
Thus, it follows that 

NY'tY. Z'TZ, 


with Z ~ S). If additionally S > 0, we may write Z = Yfl'^Z where Z ~ 

^(^- 1 / 2 ^^ /) and thus Z'TZ = Yl^s=i ^is^is where are the eigenvalues of TS 
and Xis ~ Xii^) for ^ □ 


Proof of Theorem [Z^ - The null distribution of the WTS follows analogous to the proof 
of Theorem 2.1 in |Konietsehk:e et aL] ( |2015[ ). Obviously, (pwTS is an asymptotie level a test 
and eonsistent for fixed alternatives HfifO. 

Under Hi : Hfi = it holds that y/NHY. has, asymptotieally, an N{Hu, HUH') 
distribution. Thus, the WTS has asymptotieally a non-eentral X/(^) distribution with / = 
rank{H) degrees of freedom and non-eentrality parameter 5 = {Hu)'{HYlH')^Hu. □ 


We will now proof Theorem [3T| For notational eonvenienee, we introduee 

Z = {Zm^i, . . . , ^7v,7v) = (^111) ^121, • • • , AAnil, ZlU, ■ ■ ■ , 

for the pooled sample. Sinee iTl = 0 we ean rewrite the permuted test statistie as 

Q- = i/n(y" - Y„yH'{H^"H')+y/NH(Y^ - F. 


where Y = Y 


It and Zfj = Y„ 


= ^ Ef=i Based on this representation, we 
ean split the proof of Theorem 3.1 in two results. There, we first show that the eonditional 
distribution of vW(1^. — Y f given the data is asymptotically multivariate normal. How¬ 
ever, it turns out that the resulting covariance matrix is different from S. Our approach 


corrects for the ’wrong’ covariance structure by studentizing with S , which is shown in a 
second step. Altogether, this proves the consistency of the WTPS as stated in Theorem |3.1| 
as well as the properties of the corresponding test mentioned in Remarks |3 .1 1 and [3^ 

Note that there exist finite limits bi = limmin(ni)^.oo ^ G (1, oo),i = 1,..., a because 
of (1) and 0 < maXj=i_,..^a(L) < cx). 


Lemma 8.1 Under the assumptions ofTheorem \3.1\ the conditional permutation distribu¬ 
tion of 


y/N{Y" -Y„) 
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given the observed data Y weakly converges to a multivariate normal — A^(0,cT^r) — 
distribution in probability, where 


a 

1 

2=1 


S = 1 


^ + bis 



with afg = Var(Yiks) and 


a 

r = 0 -Jt = diag{K];^It„ - Jt- 

2=1 


( 8 . 1 ) 


( 8 . 2 ) 


Proof: First note that the elassieal Cramer-Wold deviee eannot be applied direetly in this 
eontext due to the oeeurrenee of uneountably many exeeptional sets. Therefore we will 


apply a modified Cramer-Wold deviee, see e.g. the proof of Theorem 4.1 in Pauly (2011). 
Let D be a dense and eountable subset of Mf'. Then for every fixed A = (Ai,..., A^) G D 
and Mq = 0, Ml = ni,..., Mt^ = iirzi, Mi^+i = tini -f 77 , 2 ,..., Mt = N, we have 


“ /V7 

= E I) 

i=l fc=Mi_i+l * 

= 

s=i v N 


where cns = VN + 1 < s < M*}^. This implies 


Hi 


N 


{.^N,tt(s) ^ n) 


= s/nY^cms- ^ 

s=i V 


N 


s=i V 


(8.3) 


sinee it is uniformly distributed on the set of all permutations of the numbers {1,..., iV}. 
Let bi = limmin(„.)^oo ^ with bt < 00 beeause of (1) and max(fj) < 00 . 


We now apply Theorem 4.1 in Pauly (20111 to prove the eonditional eonvergenee in 
distribution. Therefore, we have to prove the following eonditions: 
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^ I p 


l<i<N 
N 


max Zjvj —0 


1 

N 


i=l 


N,i 


\2 P 2 


max |c7v,s —c. I —?• 0 

l<s<N 

N 


J2^cn,s - c.f ^i = Yl 


S = 1 


2=1 


. 2=1 


^{cnmi) - c.) 4 with E{W) = 0 and Var{W) = al 


(8.4) 

(8.5) 

( 8 . 6 ) 

(8.7) 

( 8 . 8 ) 


Condition ( |8.4| ) as well as ( |8.6[ ) - ( |8.8[ ) follow analogous to [Pauly et al.| ( |20l5a| ): Since the 
random variables within each of the a groups are i.i.d. with finite variance, they fulfill ( |8.4| ). 

The convergence in ( |8.6[ ) is obvious and since • c. = we have 


S=1 


p 


2=1 


. 2 = 1 


Moreover, (8.8) holds due to 


, 2=1 


J2{cN,--cy- = E4.-(vTcv = ^-a?- 

5=1 i=i 

T / 


P (^x/iVc„,a, _ _ j _ , ^ - 


for 1 < i < a, i.e., for a random variable W with P{W = biXi) = 1, 1 < f < a, we have 


T 


V7V(c^,^(i) -c.)AW-J2>^^ = W, 


2=1 


where W fulfills E(W) = 0 and Var{W) = a^. It remains to prove ( |8.5[ ) 


TV TV 

i=l i=l 
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Consider 


N 




^ 1 
1=1 


N,i 


^ d t-[ Th'i 

j=l S=1 k=l 


-- fl. i'j TT.'^ 

iEEE(-i+'*?.) 

j=l S=1 fc=l 

i: I E(-?.+'*?.) 

i=l s=l 


i=l * s=l 


Furthermore: 


AT 




2=1 
a ti Tii 


d ti TLi 


= »'-bEEEy“-) + CbEEE''«- 


2 = 1 S=1 fc = l 

■ V' 


N 


i=l s=l k=l 


—> 


d ^ ti 

E^E- 

^ 2 = 1 s=l 


Since 


(8.9) 


1 


N 




2 -| 22j 

^“HEiEE^.L-E 

.2=1 S=1 /C = l 


rii 


ti 


1 ^2 


52 7^ 52 ^«n 52(^2. - : 

^ (Ny ^ ■ 


. S = 1 


= o { — 

\N 

because of independence and condition (2), the desired conclusion follows with Tschebyscheff’s 
inequality. 


Altogether, this implies by Theorem 4.1 in Pauly (2011) convergence in distribution 
given the data Y 


VnX'{Y" - YJ 4 A^(0, a^al) 


( 8 . 10 ) 
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in probability. This convergence holds for every fixed X E D. Applying the subsequen- 
tial principle for convergence in probability we can find a common subsequence such 
that ( |8.10| ) holds almost surely for all A G T* along this subsequence. Now continuity 
of the characteristic function of the limit and tightness of the conditional distribution of 
\fN{Y, — Y ) given Y show that ( |8.10[ ) holds almost surely for all A G MJ’ along this 
subsequence. Thus, an application of the classical Cramer-Wold device together with an¬ 
other application of the subsequence principle imply the result. □ 


Now we will study the convergence of S = NV = 0“=^ • 


Lemma 8.2 Under the assumptions of Theorem \3. l\ we have convergence in probability 

4 4 ..., 

as N ^ oo. 

Proof: It suffices to show that (Vj —)■ a l{r = s| in probability for all 1 < r, s < U. 

Therefore consider 

i / j ^ ikr ^ iks ^ i-s 


rii 


rii 


k=l 


First, consider B. It holds: 


-V-" 

A 




B 




i=l ^ S=1 


for all r and all i analogous to ( |8.9| ). Furthermore, setting := l{(r — l)ni -f 1 < s < 


rui]/ui for 1 < s < and using Theorem 3 from Hajek et al. ( 1999) we get convergence 
in probability of the corresponding conditional variance 


N 


Var{YlfY) = Var \Y,d^S,sZN,.is)\Z 


S = 1 


N 


s=l 


(9 779 ) \ ^ 1 


N 


E 


N - 1 


Y.(Zn.. - Zf,f 4 0, 


S = 1 

since {d^S,s “ ^£.) 0 as cx) and = Op{l). 

Altogether this implies convergence in probability by the continuous mapping theorem 

2 


d ^ ti 




bi. 


i=l ‘ s=l 
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For part A we distinguish two cases: First, assume r = s. We have 


A = — 


Ui 


T. 


N,'K{k) * 


k=(r-l)ni+l 


Now consider the conditional expectation of A 


as well as 


^ Q, t-i ^ TTl'i 

i=l r=l * k={r-l)ni+l 

- E^E(-?.+/<?.) 

i=l s=l 


N 


Var{A\Y) = Vir E4!.Z«,(.)|Z 


S = 1 


N 


/ , I ^N,s ^N, 


1 


N 


N 


S=1 


N-1 




2 

N,s 


s=l 


N-1 




r=l 


which converges to 0 in probability as above and since we have 

Ylf=i (^^N,s - Ylf=i ^N,r) = C’p(l) because of the existence of fourth mo¬ 

ments. 

Now, consider r ^ s. We have that 

, rii 

E(A\Y) = -Y.E(y^ry’k.\Y) = E(Y^,,Y’,^\Y)^ 

k=l 

^ ^iV,7r(l)^7V,7r(2) 

Consider E{ZM,iZM,j)- There are two possibilities: If ZN,i and Z^j stem from different 
random vectors (i.e., from different individuals) they are independent and we can write 
E{ZN,iZNj) = E{Zf^^i)E{Zf^j). If they stem from the same individual, we cannot rewrite 
the expectation and we denote it by 7 *^ := E{ZN^iZ]sfj) G (—cxd, cxd). For every fixed i 
there are (f* — 1) possible j’s such that Z^^i and Z^j come from the same individual. This 
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implies: 




.x Y1 Ei^N,i)E{ZN,j) (8.11) 

^ ^ (M)eH 


Af 


+ 


E E 


iV(Ar- 1 ) ^ , 

^ ^ *=1 (*j) 6 yl 


where the index sets are defined as E = {{i,j,) : i ^ j and i, j stem from different subjeets} 
and A = {{i,j) i ^ j and i,j stem from the same subjeet}. 

Because of the Cauchy-Schwarz inequality and Condition (2) it holds that 


sup | 7 jj| < 2supi?(Z^ j) <C<oo. 
i,j i 

Thus, it follows that 


1 

N{N-1) 


^ 


i=l {i,j)eA 


Njmaxti - 1 ) ^ ^ ^ 
iV(iV- 1) 


as —)■ oo. For the first summand on the right hand side in Equation (8.11), it holds that 


1 

N{N-1) 


E{Z]\f^i)E{Z]\fj) = 


-> 


V / 2=1 s=l 7=1 r=l 



0 ( 1 ) 


To complete the proof it remains to show that Var EN,iZN,jJ —^ 0. Thus, 


Var 




ZN,iZN,j 


1 

(iV(iV-l))2 


ilT^jl *2^i2 


As above, we distinguish between the cases {i,j) G H and {i,j) G A. If and 

Zn^i^Znj^ stem from different individuals it holds that C'ov(Z7v,u-^iVjn ■Z^iv,i 2 -^iv.i 2 ) = 0 
because of independence. In all other cases it holds that 

Cov{ZN,nZN,j^, ZN,i 2 ZN,j 2 ) < 2supE(2'^j) = C < oo 

i 

because of assumption (2) and the Cauchy-Schwarz inequality. 

Furthermore, for every fixed ii and ji there are less than 5 (max fj)^ possibilities for Zpj^i^Zpfj^ 
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to stem from the same individual(s) as sueh that at least one of the sums eaneels 

out and Var 0 for all i, j as N ^ oo. 

This implies that for r ^ s 



and for r = s we have [V^ )r^s • Altogether, this proves the desired result. □ 

We are now able to prove Theorem [3T] 

Proof: Applying the eontinuous mapping theorem together with Lemma [8T| yields eondi- 
tional eonvergenee in distribution given Y 

hVn{Y^ -YJ 4 N{0,a^HDH'), 

where D := diag{KY I ti, ■ ■ ■ Moreover, we have eonvergenee in probability 

HJYH' 4 a^HDH' 


by Lemma 8.2 Sinee det(Vj) > 0 almost surely for N large enough due to S > 0, the 


eorresponding Moore-Penrose inverse eonverges as well in probability and henee another 
applieation of the eontinuous mapping theorem proves the result using Theorem 9.2.2 in 
Rao and Mitral (1 197 lb. □ 


9 Other resampling approaches 

9.1 Nonparametric bootstrap approach 

Here, we eonsider a nonparametrie bootstrap sample Y* = ..., Yf^Y drawn with 

replaeement from the pooled observation veetor Y = (Lm,..., YanY- Therefore, given 
the observations, the bootstrap eomponents are all independent with identieal distribution 
whieh is given by the empirieal distribution of Y*. The WTS of the bootstrap sample is 
given by 


Q% = N{Y*)'H\H%*H'YHY\ 

->(s ^ 

where Y is the veetor of means of the bootstrap sample and S denotes their eovarianee 
matrix. 
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Theorem 9.1 The distribution of Q*^ conditioned on the observed data Y weakly con¬ 
verges to the central x} distribution in probability, where f = rank{H). In particular, we 
have 


sup \Pfj.{Q*N < x\Y) - Pfj.g{QN < a;)| -)■ 0 (9.1) 

in probability for any underlying parameter p G and Pq G Hq{H). 

Proof: The result follows analogously to the proof of Theorem 3.1 in the paper. □ 


Note, that a nonparametric bootstrap version based on drawing with replacement from 
the observation vectors as in Konietschke et al. (2015) performed considerably worse than 
the parametric bootstrap approach described below and is therefore not reported here. 

In addition, we have also studied a nonparametric bootstrap version of the ATS (al¬ 
though this is in general not asymptotically correct) given by 


Fn = 


iV ,-, __* 

[Y )'TY 


tr(TS 


A corresponding permutation version of the ATS has not been considered since it is also 
asymptotically only an approximation. 


9.2 Parametric bootstrap approach 

We have also considered a parametric bootstrap approach as studied by, e.g. Konietschke 
et al. (2015). Here, the parametric bootstrap variables are generated as 

Ar(o, Vi), l<i<a. 

The idea behind this approach is to obtain a more accurate finite sample approximation by 
mimicking the given covariance structure of the original data. We can again compute the 
WTS and ATS from the parametric bootstrap vectors as 

= N(Y*,yH'{H%*HyHY*, 


and 




N 


tr(TS 


(Y*)'TY 


-★ 'k 

where Y is the vector of means of the parametric bootstrap sample and S denotes their 
empirical covariance matrix. 


30 



Theorem 9.2 The distribution of conditioned on the observed data Y weakly con¬ 
verges to the central xj distribution in probability, where f = rank{H). In particular, we 
have 


sup < x\Y) - Pfj,g{QN < a;)| -^0 (9.2) 

in probability for any underlying parameters p, Pq G with Hpq = 0. 

Furthermore, for the ATS of the parametric bootstrap sample it also holds that 

sup \P^J,{Ff < x\Y) - P/xo(^ 7 v < a:)| -)■ 0 (9.3) 

in probability for any underlying parameters p, Pq G with Hp^ = 0. Thus, the 
conditional distribution of Ff always approximates the null distribution of F^- 

Proof: The result for the WTS follows analogously to the proof of Theorem 3.2 in Koni- 
etsehke et al. (2015). For the parametrie bootstrap version of the ATS the result is obtained 
by the multivariate Lindeberg-Feller Theorem, the Continuous Mapping Theorem and an¬ 
other application of Slutsky’s Theorem. The details are left to the reader. □ 


9.3 Type-I error rates 


In the following, we present the results of the detailed simulation studies conducted as 
described in Section of the paper. For comparison, the results of the permutation ap¬ 
proach are also included. The results for the hypothesis of no time effect T are presented 
in Tables [^1^ and for the normal, log-normal and exponential distribution, respectively. 
The results for the hypothesis of no group x time interaction are in Tables 10 and[TT| 
respectively. The parametric bootstrap approach is denoted by PBS, the nonparametric 
bootstrap by NPBS. The results are again compared to the asymptotic quantiles, i.e. the 
F(z/, oo)-quantile for the ATS and the X/-quantile for the WTS. A permutation version of 
the ATS has not been considered for the reasons stated above. The covariance settings and 
the number of simulated individuals are the same as described in Section |4l 
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Table 6: Simulation results for the hypothesis of no time effect with normal distribution. 

normal distribution 


T 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.041 0.052 

0.048 0.047 

0.046 0.085 

NA 0.050 
0.034 0.059 

0.052 0.050 

0.040 0.177 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.048 
0.041 0.050 

0.046 0.048 

0.046 0.086 

NA 0.052 

0.034 0.060 

0.052 0.050 

0.040 0.177 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 
0.045 0.048 

0.051 0.051 

0.050 0.078 

NA 0.052 
0.040 0.050 

0.050 0.052 

0.043 0.135 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.046 0.052 

0.056 0.051 

0.051 0.085 

NA 0.051 

0.036 0.059 

0.054 0.050 

0.042 0.177 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 
0.046 0.052 

0.057 0.051 

0.052 0.086 

NA 0.052 

0.036 0.060 

0.056 0.052 

0.043 0.177 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 

0.049 0.048 

0.059 0.049 

0.053 0.077 

NA 0.052 

0.038 0.049 

0.054 0.051 

0.041 0.135 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.052 
0.041 0.052 

0.052 0.052 

0.046 0.092 

NA 0.062 
0.040 0.064 

0.069 0.061 

0.044 0.198 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.045 
0.047 0.052 

0.056 0.043 

0.051 0.080 

NA 0.042 

0.043 0.056 

0.075 0.042 

0.048 0.155 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.053 
0.047 0.050 

0.058 0.051 

0.051 0.078 

NA 0.054 

0.044 0.049 

0.073 0.052 

0.048 0.136 
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Table 7: Simulation results for the hypothesis of no time effeet with log-normal distribution. 


log-normal distribution 


T 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 

0.026 0.055 

0.051 0.050 

0.032 0.094 

NA 0.047 

0.017 0.075 

0.047 0.048 

0.021 0.198 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.052 

0.025 0.058 

0.048 0.051 

0.031 0.090 

NA 0.046 

0.016 0.074 

0.054 0.046 

0.020 0.198 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 
0.026 0.056 

0.052 0.050 

0.031 0.089 

NA 0.048 
0.019 0.077 

0.051 0.050 

0.021 0.186 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.067 

0.035 0.072 

0.060 0.066 
0.040 0.110 

NA 0.053 
0.018 0.084 

0.052 0.053 

0.022 0.207 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.067 

0.034 0.073 

0.061 0.066 
0.040 0.107 

NA 0.051 

0.018 0.082 
0.057 0.052 

0.022 0.203 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.070 

0.037 0.072 

0.065 0.068 

0.042 0.107 

NA 0.057 

0.021 0.080 
0.057 0.057 

0.024 0.197 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.057 

0.027 0.059 

0.054 0.058 

0.033 0.101 

NA 0.064 

0.021 0.082 
0.063 0.064 

0.024 0.221 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.053 
0.031 0.060 

0.057 0.053 

0.037 0.090 

NA 0.048 

0.028 0.075 

0.079 0.047 

0.033 0.190 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.057 

0.031 0.059 

0.057 0.054 

0.036 0.092 

NA 0.062 

0.027 0.079 

0.075 0.062 

0.031 0.191 
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Table 8: Simulation results for the hypothesis of no time effect with exponential distribu¬ 
tion. 


exponential distribution 


T 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.048 

0.038 0.055 

0.052 0.049 

0.045 0.090 

NA 0.051 

0.026 0.070 

0.052 0.051 

0.034 0.194 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.053 
0.039 0.057 

0.053 0.053 

0.046 0.096 

NA 0.048 

0.029 0.069 

0.050 0.048 

0.032 0.191 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 
0.041 0.057 

0.053 0.054 

0.046 0.086 

NA 0.050 
0.031 0.059 

0.049 0.050 

0.034 0.151 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 
0.040 0.059 

0.062 0.055 

0.048 0.093 

NA 0.052 

0.029 0.070 

0.057 0.052 

0.035 0.194 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.060 
0.044 0.064 

0.063 0.059 

0.050 0.101 

NA 0.051 

0.029 0.074 

0.056 0.052 

0.034 0.193 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.058 
0.045 0.062 

0.060 0.058 

0.050 0.088 

NA 0.051 

0.032 0.062 

0.052 0.051 

0.036 0.154 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.055 
0.041 0.055 

0.060 0.054 

0.049 0.098 

NA 0.066 
0.034 0.074 

0.073 0.065 

0.042 0.218 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.049 

0.045 0.058 

0.062 0.049 

0.050 0.090 

NA 0.045 

0.039 0.067 

0.078 0.044 

0.046 0.173 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.055 
0.045 0.058 

0.062 0.055 

0.050 0.087 

NA 0.056 
0.038 0.060 

0.072 0.056 

0.042 0.153 
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Table 9: Simulation results for the hypothesis of no group x time interaction with normal 
distribution. 

normal distribution 


GT 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.046 

0.039 0.051 

0.049 0.046 

0.049 0.135 

NA 0.051 

0.025 0.077 

0.052 0.051 

0.033 0.432 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.052 

0.042 0.056 

0.052 0.051 

0.053 0.142 

NA 0.050 
0.026 0.075 

0.051 0.049 

0.034 0.433 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.049 
0.041 0.049 

0.050 0.050 

0.048 0.126 

NA 0.051 
0.032 0.046 

0.054 0.050 

0.039 0.366 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.045 0.054 

0.060 0.050 

0.053 0.132 

NA 0.052 

0.030 0.076 

0.055 0.053 

0.038 0.429 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 

0.044 0.056 

0.059 0.053 

0.053 0.141 

NA 0.050 
0.029 0.072 

0.057 0.051 

0.038 0.431 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.052 

0.044 0.049 

0.059 0.052 

0.050 0.122 

NA 0.050 
0.034 0.046 

0.060 0.050 

0.040 0.366 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.043 0.051 

0.061 0.049 

0.054 0.141 

NA 0.065 
0.033 0.082 

0.075 0.069 

0.040 0.465 

n(2) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.045 

0.046 0.054 

0.057 0.047 

0.053 0.135 

NA 0.037 

0.040 0.069 

0.078 0.037 

0.049 0.393 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.049 

0.043 0.048 

0.064 0.050 

0.051 0.126 

NA 0.053 
0.038 0.047 

0.077 0.051 

0.045 0.363 
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Table 10: Simulation results for the hypothesis of no group x time interaction with log¬ 
normal distribution. 


log-normal distribution 


GT 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.047 

0.019 0.040 

0.048 0.047 

0.024 0.121 

NA 0.053 
0.009 0.061 

0.048 0.052 

0.012 0.426 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.053 
0.017 0.044 

0.048 0.053 

0.022 0.128 

NA 0.051 

0.009 0.055 

0.050 0.048 

0.013 0.431 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.048 
0.018 0.037 

0.048 0.046 

0.024 0.118 

NA 0.051 
0.010 0.042 

0.047 0.051 

0.012 0.406 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.051 
0.019 0.044 

0.056 0.051 

0.025 0.129 

NA 0.054 

0.010 0.062 
0.052 0.054 

0.014 0.427 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 
0.019 0.044 

0.056 0.053 

0.026 0.130 

NA 0.052 

0.011 0.056 

0.052 0.051 

0.013 0.432 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.018 0.038 

0.056 0.049 

0.023 0.120 

NA 0.052 

0.010 0.042 

0.053 0.052 

0.013 0.403 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.050 
0.022 0.042 

0.053 0.050 

0.029 0.133 

NA 0.062 

0.014 0.067 

0.068 0.060 
0.020 0.457 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.045 

0.022 0.043 

0.055 0.046 

0.028 0.121 

NA 0.036 
0.020 0.053 

0.076 0.035 

0.024 0.399 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.049 

0.023 0.037 

0.059 0.046 

0.028 0.122 

NA 0.053 
0.014 0.043 

0.071 0.054 

0.020 0.408 
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Table 11: Simulation results for the hypothesis of no group x time interaction with expo¬ 
nential distribution. 


exponential distribution 


GT 

t = 4 

t = 8 

Cov. Setting 

n 

Method 

ATS WTS 

ATS WTS 

1 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 

0.036 0.057 

0.055 0.055 

0.043 0.146 

NA 0.054 

0.018 0.076 

0.049 0.055 

0.024 0.442 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.054 

0.030 0.057 

0.051 0.054 

0.041 0.148 

NA 0.050 
0.019 0.072 

0.048 0.050 

0.024 0.443 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.047 
0.029 0.043 

0.047 0.046 

0.036 0.122 

NA 0.054 
0.023 0.052 

0.054 0.054 

0.028 0.397 

2 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.059 
0.040 0.061 

0.065 0.061 

0.048 0.151 

NA 0.057 

0.019 0.077 

0.054 0.056 

0.027 0.444 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.059 
0.035 0.060 

0.058 0.059 

0.042 0.153 

NA 0.052 

0.019 0.072 

0.050 0.051 

0.025 0.448 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.048 

0.028 0.042 

0.051 0.048 

0.034 0.121 

NA 0.055 
0.024 0.051 

0.058 0.054 

0.029 0.397 

3 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.061 

0.039 0.059 

0.067 0.060 

0.047 0.155 

NA 0.068 
0.024 0.083 

0.068 0.069 

0.032 0.473 


Permutation 

PBS 

NPBS 

asymptotic 

NA 0.049 

0.038 0.056 

0.056 0.050 

0.043 0.140 

NA 0.037 

0.033 0.062 

0.078 0.036 

0.042 0.406 

n(3) 

Permutation 

PBS 

NPBS 

asymptotic 

NA 0.047 

0.031 0.043 

0.055 0.047 

0.037 0.122 

NA 0.058 
0.033 0.052 

0.080 0.057 

0.041 0.402 
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10 Additional simulation results: Quality of the approxi¬ 
mation 

Recall that we defined 


KQS= sup \F^\t) - F-\t)\ 


0.9<t<0.99 


as well as 


KQS-= sup 

0.9<t<0.99 


for the distances between the quantile functions of the WTS (Fjy ) and the x^-distribution(F 
and the WTPS((F^)“^), respectively. The results for all simulation settings described in 


the paper are presented in Tables 12 and 13 A plot of one exemplarily chosen scenario is 
in Figure]^ 



Quantile Function 


Figure 3: Quantile funtions of the WTS, WTPS and the corresponding yf-distribution in 
the simulation setting with normally distributed data, t = A, covariance matrix setting 3, 
and under the null hypothesis of no time effect. 
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Table 12: Simulation results for the distanees between the quantile functions for the hy¬ 
pothesis of no time effect. 

normal distribution 


T 

t 

= 4 

t = 

8 

Cov. Setting 



WTPS 


WTPS 



3.683 

0.411 

10.548 

0.381 

1 

n''‘^ 

3.299 

0.310 

11.393 

0.654 


nC-i) 

2.198 

0.213 

7.771 

0.281 



3.620 

0.564 

10.494 

0.286 

2 


3.378 

0.227 

11.604 

0.993 


nC-i) 

1.991 

0.226 

7.646 

0.297 



4.451 

1.186 

12.515 

2.086 

3 


2.599 

0.731 

9.564 

1.486 



2.264 

0.105 

7.571 

0.344 


log-normal distribution 


T 

t 

= 4 

t = 

8 

Cov. Setting 



WTPS 

X^ 

WTPS 


nC) 

3.097 

0.532 

12.399 

1.044 

1 


3.960 

0.386 

14.293 

0.998 


nC-i) 

3.087 

0.363 

12.768 

0.421 



5.645 

2.258 

14.165 

1.105 

2 


6.045 

2.656 

15.484 

2.617 


nC-i) 

5.062 

1.891 

14.239 

1.815 



3.977 

0.517 

14.547 

2.299 

3 


4.000 

0.526 

12.740 

0.610 


n}-'A) 

3.561 

0.643 

14.238 

3.203 


exponential distribution 


T 

t 

= 4 

t = 

8 

Cov. Setting 


X^ 

WTPS 

X^ 

WTPS 


nC) 

3.617 

0.283 

11.750 

1.054 

1 

nC^) 

4.245 

0.491 

12.098 

0.948 


nC-^) 

2.906 

0.382 

9.685 

0.885 



4.761 

1.262 

11.704 

0.628 

2 


4.961 

1.366 

12.201 

0.724 


nC-i) 

3.833 

0.915 

10.226 

0.286 



4.567 

0.969 

13.840 

2.089 

3 


3.700 

0.290 

10.504 

1.729 


nC-i) 

3.179 

0.343 

10.093 

0.601 
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Table 13: Simulation results for the distanees between the quantile funetions for the hy¬ 
pothesis of no interaction. 

normal distribution 


GT 

t = 

4 

t = 

8 

Cov. Setting 



WTPS 


WTPS 


nl 

9.151 

0.420 

40.954 

1.135 

1 

n2 

8.872 

0.573 

41.179 

1.451 


n3 

7.789 

0.711 

30.617 

0.905 


nl 

8.648 

0.582 

42.023 

1.804 

2 

n2 

8.727 

0.497 

41.980 

1.928 


n3 

7.951 

1.166 

30.031 

0.956 


nl 

10.280 

1.108 

48.106 

7.618 

3 

n2 

7.700 

1.463 

36.252 

4.470 


n3 

7.579 

0.604 

31.374 

1.461 


log-normal distribution 


GT 

t 

= 4 

t = 

8 

Cov. Setting 



WTPS 

X^ 

WTPS 


nl 

5.292 

0.610 

30.474 

1.058 

1 

n2 

5.952 

0.360 

30.014 

1.671 


n3 

5.767 

0.467 

27.019 

1.024 


nl 

5.340 

0.524 

30.986 

0.770 

2 

n2 

6.307 

0.812 

29.960 

1.329 


n3 

5.826 

0.674 

27.346 

0.558 


nl 

6.425 

0.298 

34.755 

2.106 

3 

n2 

5.124 

1.408 

26.657 

6.691 


n3 

5.561 

0.182 

27.517 

1.363 


exponential distribution 


GT 

t 

= 4 

t = 

8 

Cov. Setting 


x^ 

WTPS 

X^ 

WTPS 


nl 

8.416 

0.431 

36.706 

0.968 

1 

n2 

9.066 

1.184 

37.318 

1.295 


n3 

6.016 

0.618 

29.863 

1.073 


nl 

8.523 

0.869 

36.999 

1.044 

2 

n2 

9.206 

1.510 

37.160 

1.436 


n3 

6.445 

0.293 

30.130 

0.925 


nl 

9.415 

1.219 

42.643 

5.264 

3 

n2 

7.638 

0.946 

32.131 

5.851 


n3 

6.490 

0.260 

30.012 

0.689 


11 Power 


We have also conducted several simulations to analyze the power of our method. Since 
the WTS turned out to test on different ct—levels (see the simulation results under the null 
hypothesis), we have excluded it from the analyses. We considered a two sample repeated 
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measures design, where we have simulated data as 

Yik = {Yiki,..., Yikt)' = + V J Sik, 

with = E{Yii),i = 1, 2, and V* = It- Thei.i.d. random veetors = (ejfci, • • •, eikt),i = 
1,2, were generated from different standardized distributions by 


_ (^iks E(^6iks') 

where liks denote i.i.d. normal or log-normal random variables. For the power simula¬ 
tion we have eonsidered a trend alternative, i.e. we set ^2 = ^ the seeond group 
and = 6c = 6{ci ,..., q)' in the first group, where Cs = f, s = 1,..., f and 6 G 
{0, 0.5,1,1.5,2, 3}. We eonsidered a balaneed design with 15 individuals per group, hy¬ 
pothesis matrix H = — It) and again simulated both t = A and t = 8 repeated 

measures. Figures]^ and [^display the power eomparison for the WTPS, the ATS, the ap¬ 
proximation deseribed by Leeoutre (1991) as well as Hotelling’s (Hotelling, 1931) for 
normal distribution and t = A and t = 8 repeated measures, respeetively. In Figures and 
1^ the results for the log-normal distribution are displayed. From these figures it appears 
that the ATS has slightly higher power for normally distributed data. For log-normally dis¬ 
tributed data, the WTPS has larger power than the other methods and it is the only method 
eontrolling the type-I error eorreetly. We also note that the approximation by Huynh-Feldt 
and Leeoutre performs worst for the log-normal distribution. 



Delta 


Figure 4: Power (a = 0.05) simulation results of the WTPS, ATS, Leeoutre and Hotelling 
for normal distribution and t = A repeated measures under a trend alternative p = 
{pfp' 2 )' = (O', Sd)' with 6 = {0,0.5,1,1.5, 2, 3} and = |, s = 1,..., f. 
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Delta 


Figure 5: Power (a = 0.05j simulation results of the WTPS, ATS, Lecoutre and Hotelling 
for normal distribution and t = 8 repeated measures under a trend alternative p = 
{pf P 2 y = (o', 6c'y with S = {0, 0.5,1,1.5, 2, 3} and = |, s = 1,..., t. 



Delta 


Figure 6: Power (a = 0.05j simulation results of the WTPS, ATS, Lecoutre and Hotelling 
for log-normal distribution and t = 4 repeated measures under a trend alternative p = 
{pf p^y = (o', 5c'y with 5 = {0, 0.5,1,1.5, 2, 3} and = |, s = 1,..., t. 
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Delta 


Figure 7: Power (a = 0.05j simulation results of the WTPS, ATS, Lecoutre and Hotelling 
for log-normal distribution and t = 8 repeated measures under a trend alternative p = 
{pfp' 2 )' = (O', dd)' with 6 = {0,0.5,1,1.5, 2, 3} and Cg = |, s = 1,..., t. 


12 Analysis of the data example: Comparing the different 
approaches 


We again consider the data example from Section]^ on the oxygen consumption of leuko¬ 
cytes. First of all, we notice that the empirical covariance matrices of the two groups ap¬ 
pear to be quite different. The empirical covariance matrix in the Placebo-group (rounded 
to three digits) is given as 


0.025 

-0.022 

-0.004 

0.009 

0.015 

0.025 

-0.022 

0.092 

-0.005 

-0.001 

-0.024 

-0.035 

-0.004 

-0.005 

0.081 

-0.013 

-0.010 

-0.004 

0.009 

-0.001 

-0.013 

0.037 

0.044 

0.038 

0.015 

-0.024 

-0.010 

0.044 

0.069 

0.063 

0.025 

-0.035 

-0.004 

0.038 

0.063 

0.115 


whereas in the Verum-group we have 
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/ 

0.043 

0.012 

0.046 

0.033 

0.014 

0.055 

\ 


0.012 

0.113 

0.008 

0.009 

0.060 

0.032 



0.046 

0.008 

0.065 

0.041 

0.005 

0.066 



0.033 

0.009 

0.041 

0.047 

0.016 

0.059 



0.014 

0.060 

0.005 

0.016 

0.058 

0.047 


V 

0.055 

0.032 

0.066 

0.059 

0.047 

0.116 

/ 


Thus, the assumption of homoscedasticity is not fulfilled in this data example. 

The results of the analyses using the different methods are presented in the follow¬ 
ing table. The asymptotie results are again obtained by eonsidering the corresponding 
F(z>, cx))-quantile for the ATS and the X/-quantile for the WTS. 


Table 14: p-values of the analysis of the O 2 consumption data. 



ATS 

WTS 


asymptotic 

PBS 

NPBS 

asymptotic 

Permutation 

PBS 

NPBS 

A 

0.001 

0.003 

<0.001 

0.001 

0.002 

0.004 

0.005 

B 

<0.001 

0.001 

0.001 

<0.001 

<0.001 

0.001 

0.001 

T 

<0.001 

<0.001 

<0.001 

<0.001 

<0.001 

<0.001 

<0.001 

AB 

0.110 

0.130 

0.140 

0.110 

0.118 

0.125 

0.136 

AT 

0.009 

0.012 

0.005 

<0.001 

0.001 

<0.001 

<0.001 

BT 

0.094 

0.088 

0.103 

0.115 

0.147 

0.161 

0.157 

ABT 

0.117 

0.154 

0.116 

0.116 

0.162 

0.153 

0.141 


For this data set, the results are similar for all resampling methods and the asymptotic 
approaches considered. 
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